Skip to content

Uniformize monorepo structure to OrdinaryDiffEq canonical style#616

Merged
ChrisRackauckas merged 8 commits into
SciML:masterfrom
ChrisRackauckas-Claude:uniformize-monorepo-structure
Jun 8, 2026
Merged

Uniformize monorepo structure to OrdinaryDiffEq canonical style#616
ChrisRackauckas merged 8 commits into
SciML:masterfrom
ChrisRackauckas-Claude:uniformize-monorepo-structure

Conversation

@ChrisRackauckas-Claude

Copy link
Copy Markdown
Contributor

Uniformizes DataDrivenDiffEq.jl to the OrdinaryDiffEq.jl canonical monorepo style: the lib/* sublibraries become a real path-graph project model tested through the centralized SublibraryCI/DowngradeSublibraries workflows, and the root track is reduced to the root package's own groups.

Canonicalization done

[sources] path graph (the MAJOR fix)

  • Each sublibrary (DataDrivenDMD, DataDrivenSR, DataDrivenSparse, DataDrivenLux) gains [sources] DataDrivenDiffEq = {path = "../.."}. The sublibraries are leaves that depend on the in-repo root DataDrivenDiffEq; without [sources] they resolved against the registered release instead of the PR checkout. The root does not re-export the sublibraries (no sublib deps in root [deps]), so per the true dependency direction no cyclic root→leaf [sources] edges were added.

Single group env var (DATADRIVENDIFFEQ_TEST_GROUP) threaded consistently

  • SublibraryCI.yml: with: { group-env-name: DATADRIVENDIFFEQ_TEST_GROUP, check-bounds: auto }, secrets: inherit.
  • DowngradeSublibraries.yml: group-env-name: DATADRIVENDIFFEQ_TEST_GROUP, group-env-value: Core; no allow-reresolve (v1 removed that input); skip list = stdlibs + the path-developed sibling DataDrivenDiffEq; sublibraries auto-discovered.
  • Every sublibrary test/runtests.jl now reads get(ENV, "DATADRIVENDIFFEQ_TEST_GROUP", get(ENV, "GROUP", "All")) and runs its functional suite on Core/All. DataDrivenSparse and DataDrivenSR previously had no group gate at all.

Root track

  • CI.yml: dropped the hardcoded sublibrary GROUP rows (DataDrivenDMD/SR/Sparse/Lux) so sublibraries are tested only through SublibraryCI's project model and are not double-tested; kept a root-only group set (Core, nopre); coverage scoped to src.
  • Downgrade.yml: group: Core, no allow-reresolve; added a concurrency block.
  • test/runtests.jl: replaced the ad-hoc dispatcher with an OrdinaryDiffEq-style _detect_sublibrary_group dispatcher — if GROUP names lib/<X> (optionally lib/<X>_<TESTGROUP>) it activates the sublibrary, develops its [sources] transitively on Julia < 1.11, sets DATADRIVENDIFFEQ_TEST_GROUP, and Pkg.tests it; otherwise it runs the root @safetestset groups (All/Core/Downstream) or the nopre JET env.

test_groups.toml

  • Added [Core] on ["lts","1.11","1","pre"] to each sublibrary so the default QA group (no QA/Aqua/JET suite exists in these sublibraries) is not spawned.

Invariants preserved

  • No [compat] floor lowered; all julia = "1.10" floors kept.
  • No test skipped/@test_broken/@test_skip/commented-out; no tolerance loosened.
  • No bare test/Project.toml exists (the per-group test/nopre/Project.toml is the canonical isolated-env pattern, kept as-is — same as OrdinaryDiffEq's test/qa/Project.toml etc.).
  • No ReTestItems/@testitem present anywhere (the repo already used SafeTestsets).

Verified locally (isolated depot, Julia 1.10 + 1.11)

  • TOML-parsed every changed Project.toml; confirmed every [targets].test dep appears in [extras]/[deps]; confirmed each sublib has [sources] DataDrivenDiffEq = {path="../.."}; parsed all four test_groups.toml.
  • YAML-parsed CI.yml, SublibraryCI.yml, DowngradeSublibraries.yml, Downgrade.yml.
  • Instantiated the root and a representative sublibrary; [sources] resolves the in-repo root natively on 1.11.
  • Ran the root Core group (Julia 1.10): All | 260 260Testing DataDrivenDiffEq tests passed.
  • Ran DataDrivenSparse Core via the faithful SublibraryCI path (Pkg.test + DATADRIVENDIFFEQ_TEST_GROUP=Core, --check-bounds=auto, Julia 1.11, [sources] native): Basic Sparse Regression 33/33, Pendulum 48/48, Michaelis Menten 9/9, Cartpole 4/4Testing DataDrivenSparse tests passed.
  • Confirmed the QA group no-ops cleanly for a sublibrary (Pkg.test passes with zero test sets).
  • Ran Runic on all changed .jl; all clean.

What CI must confirm (not run locally)

  • The other three sublibraries (DataDrivenDMD, DataDrivenLux, DataDrivenSR) under SublibraryCI across lts/1.11/1/pre (only DataDrivenSparse was run locally as the representative).
  • The centralized sublibrary-project-tests.yml@v1 matrix discovery / coverage upload.
  • DowngradeSublibraries and Downgrade downgrade resolution.
  • The root nopre (JET) group.

Ignore until reviewed by @ChrisRackauckas.

ChrisRackauckas and others added 8 commits June 7, 2026 16:01
Make the lib/* sublibrary project model real and route sublibrary testing
through the centralized SublibraryCI/DowngradeSublibraries workflows, matching
the OrdinaryDiffEq.jl canonical monorepo layout.

- Add [sources] DataDrivenDiffEq = {path = "../.."} to each sublibrary
  (DataDrivenDMD/SR/Sparse/Lux). The sublibraries are leaves that depend on the
  in-repo root DataDrivenDiffEq; without [sources] they resolved against the
  registered release instead of the PR checkout. Root does not re-export the
  sublibraries, so no cyclic root->leaf [sources] edges are added.

- Thread a single group-env-name (DATADRIVENDIFFEQ_TEST_GROUP) consistently:
  - SublibraryCI.yml: group-env-name + check-bounds: auto.
  - DowngradeSublibraries.yml: group-env-name/group-env-value: Core
    (no allow-reresolve; v1 removed that input). DataDrivenDiffEq is skipped
    (path-developed sibling) alongside stdlibs.
  - Every sublibrary test/runtests.jl now reads
    get(ENV, "DATADRIVENDIFFEQ_TEST_GROUP", get(ENV, "GROUP", "All")) and runs
    its functional suite on Core/All; DataDrivenSparse/SR previously had no
    group gate.

- Root CI.yml: drop the hardcoded sublibrary GROUP rows (DataDrivenDMD/SR/
  Sparse/Lux) so sublibraries are tested only via SublibraryCI's project model;
  keep a root-only group set (Core, nopre). Coverage scoped to src.

- Root Downgrade.yml: group: Core, no allow-reresolve.

- Root test/runtests.jl: replace the ad-hoc dispatcher with a
  _detect_sublibrary_group dispatcher (OrdinaryDiffEq style) that activates
  lib/<X>, develops its [sources] on Julia < 1.11, sets the group env var, and
  Pkg.tests the sublibrary; otherwise runs the root @safetestset groups / nopre.

- Add test/test_groups.toml ([Core] on lts/1.11/1/pre) to each sublibrary so the
  default QA group (which has no QA suite here) is not spawned.

No [compat] floors lowered; no tests skipped/broken/loosened. No test/Project.toml
exists (the per-group test/nopre/Project.toml is the canonical isolated-env
pattern and is kept). No ReTestItems/@testitem present.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Match the OrdinaryDiffEq canonical monorepo style: every sublibrary gets a
README with the standard SciML badge block (Zulip, Global Docs, ColPrac,
SciML Code Style) and the "component of the DataDrivenDiffEq.jl monorepo"
wording, plus a one-line description of the sublibrary's role and a link to
the DataDrivenDiffEq documentation.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Match the OrdinaryDiffEq canonical monorepo style, where the root and every
sublibrary ship a LICENSE.md. Content is preserved verbatim (git mv).

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
TagBot.yml: adopt the OrdinaryDiffEq canonical form -- top-level permissions
block, workflow_dispatch.inputs.lookback default "3", a root
TagBot-DataDrivenDiffEq job, and a TagBot-Subpackages matrix (fail-fast:false)
over the four registered sublibraries (DataDrivenDMD, DataDrivenLux,
DataDrivenSR, DataDrivenSparse), each tagging via subdir: lib/<Name>. All four
are registered in the General registry.

Downstream.yml (IntegrationTest): add the canonical downstream integration
workflow. Develops the root umbrella DataDrivenDiffEq plus every lib/*
sublibrary, then runs the downstream repo's tests; ResolverError is treated as
an intentional breaking change (exit 0). Matrix covers the external General
dependents SymbolicNumericIntegration.jl and ODINN.jl.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… to 1.15

Every lib/* sublibrary now has a [compat] entry for each [deps] and [extras]
package, removing the implicit-unbounded test/stdlib dependencies:

- DataDrivenDMD: add LinearAlgebra, SafeTestsets, StableRNGs, StatsBase, Test.
- DataDrivenLux: add Distributed, LinearAlgebra, Logging, Random, SafeTestsets,
  StableRNGs, Test.
- DataDrivenSparse: add LinearAlgebra, Printf, Random, SafeTestsets,
  StableRNGs, StatsBase, Test.
- DataDrivenSR: add StableRNGs, StatsBase, Test.

Stdlib bounds track the julia = "1.10" floor ("1.10"); SafeTestsets = "0.1",
StableRNGs = "1", StatsBase = "0.34".

Bump the DataDrivenDiffEq compat floor in each sublibrary from "1.14" to "1.15"
to match the in-repo umbrella version (1.15.0). All four sublibraries resolve
against the in-repo umbrella via their [sources] path entries (verified with
Pkg.resolve on Julia 1.11). No floors lowered; only additions and the
umbrella-matching raise.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the missing [compat] entries so every root [deps] and [extras] package is
bounded, matching the canonical fully-bounded monorepo style: LinearAlgebra,
Random, Pkg, Test = "1" (stdlibs, consistent with the existing Statistics = "1")
and SafeTestsets = "0.1". Root environment resolves cleanly with these bounds
(verified with Pkg.resolve on Julia 1.11). Only additions; no floors lowered.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Move every test group's files into test/<Group>/ folders, per the
OrdinaryDiffEq canonical dependency-driven group model, at both the root
package and each lib/* sublibrary.

Root (DataDrivenDiffEq):
- Functional tests (basis, problem, solution, commonsolve, utils) move
  from their scattered subfolders into test/Core/; runtests.jl includes
  them from Core/. Core is the base-env group and the only group in All.
- The nopre group (JET static analysis) already lives in test/nopre/ with
  its own Project.toml (JET) — a dep-adding group excluded from All. Its
  name stays lowercase to match the lowercase nopre input in CI.yml.
- Add test/test_groups.toml declaring Core [lts,1,pre] and nopre [lts,1].

Sublibraries (DataDrivenDMD, DataDrivenLux, DataDrivenSparse, DataDrivenSR):
- Each runs only a functional Core group (no QA tooling, no GPU). Move
  the Core test files into lib/<sublib>/test/Core/ and include from there.
  DataDrivenSR's inline runtests body is extracted to
  Core/symbolic_regression.jl. Core adds no deps beyond the base test set,
  so no per-group Project.toml is introduced.
- Normalize each test_groups.toml Core version matrix from
  [lts,1.11,1,pre] to the canonical [lts,1,pre].

No compat floors lowered, no tests skipped/broken/loosened; every test
that previously ran still runs in exactly one group.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Replace TagBot.yml with the canonical thin caller to
  SciML/.github tagbot.yml@v1, plus a tagbot-subpackages matrix over
  the registered lib/* sublibraries (DataDrivenDMD, DataDrivenLux,
  DataDrivenSparse, DataDrivenSR; all confirmed in General).
- Downgrade.yml: drop hand-listed stdlib skip (Pkg,TOML) and the
  julia-version: "1.10" pin (central default is lts; auto-populated skip).
- DowngradeSublibraries.yml: drop hand-listed skip (stdlibs + caller
  package), now auto-populated by the centralized workflow.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@ChrisRackauckas-Claude ChrisRackauckas-Claude marked this pull request as ready for review June 8, 2026 09:39
@ChrisRackauckas ChrisRackauckas merged commit 9e38dfd into SciML:master Jun 8, 2026
14 of 15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants